Wireless wrist computing and control device and method for 3D imaging, mapping, networking and interfacing

ABSTRACT

An apparatus and method for light and optical depth mapping, 3D imaging, modeling, networking, and interfacing on an autonomous, intelligent, wearable wireless wrist computing, display and control system for onboard and remote device and graphic user interface control. Embodiments of the invention enable augmentation of people, objects, devices and spaces into a virtual environment and augmentation of virtual objects and interfaces into the physical world through its wireless multimedia streaming and multi-interface display and projection systems.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority to provisional application Ser. No.61/796,056, filed Nov. 1, 2012, which application is incorporated hereinin its entirety by this reference thereto.

TECHNICAL FIELD

This invention relates to the field of wearable computing. Moreparticularly, the invention relates to devices and methods for 3Dmapping, imaging, networking, communications, and multi-interface remotecontrolling.

DESCRIPTION OF RELATED ART

Existing control devices include the early mouse technology in the formof hand-held wired x-y positional input devices, such as found in U.S.Pat. No. 3,541,541, to sensor and spatial positioning systems such asU.S. Pat. No. 6,005,548, to wearable optical hand, finger and objectspatial positioning systems that incorporate gesture and voicerecognition and touchscreen interfacing controls.

Prior art such as U.S. Pat. No. 6,647,632 introduced a wireless controldevice worn on the wrist with light emitters and sensors placed on theinside of the hand to identify the position of the users hand andfingers, recognize pre-assigned gestures and voice commands and relaythe data to a controlled device and U.S. Pat. No. 8,292,833 B2introduced a wrist worn Finger Motion Detecting Apparatus that usesoptical and ultrasonic wave signal monitoring of the wearers tendons toidentify the position and movement of their hand and fingers and relaydata to a controlled device. U.S. Patent Application 2009/0096783 A1introduces an indoor three dimensional structured imaging system andbody motion and gesture interfacing system using a light speckle patternto 3D map illuminated objects and U.S. Patent Application 2011/0025827introduces stereoscopic depth mapping using a combination of lightprojection and 3D color imaging, both systems are limited to depthmapping, modeling and interfacing from a fixed location.

A common attribute of the mouse and other handheld and wearableinterfacing devices is the definition of the controllers beingperipheral devices, and a positional data input accessories to remotecontrolled devices and computing systems. Therefore, there are manyproblems with the known with existing technology.

SUMMARY OF THE INVENTION

An apparatus and method for light and optical depth mapping, 3D imaging,modeling, networking, and interfacing on an autonomous, intelligent,wearable wireless wrist computing, display and control system foronboard and remote device and graphic user interface control.Embodiments of the invention enable augmentation of people, objects,devices, and spaces into a virtual environment and augmentation ofvirtual objects and interfaces into the physical world through itswireless multimedia streaming and multi-interface display and projectionsystems.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a perspective view of an embodiment of the wrist console.

FIG. 1B is a perspective view of the embodiment shown in FIG. 1A.

FIG. 1C is a perspective view of the embodiment shown in FIG. 1A.

FIG. 1D is a top plan view of the embodiment shown in FIG. 1A.

FIG. 2A is a perspective view of the embodiment shown in FIG. 1Adepicting an expansion module.

FIG. 2B is a perspective view of the embodiment shown in FIG. 1Adepicting the detached expansion module.

FIG. 2C is a perspective view of the embodiment shown in FIG. 1Adepicting an expansion.

FIG. 2D is a perspective view of the embodiment shown in FIG. 1A withthe expansion module detached.

FIGS. 3A and 3B are perspective views illustrating an embodiment of thewrist console that incorporates one or more moving beams of light.

FIGS. 3C and 3D are perspective views depicting an embodiment of theinvention that incorporates structured light imaging.

FIGS. 3E and 3F are perspective views illustrating a wrist consolegenerating a depth and color mapped hand.

FIGS. 3G and 3H are perspective views illustrating a wrist consoleidentifying the precise position of joints and creases of the hand.

FIGS. 3I and 3J are perspective views illustrating a wrist consolegenerating a functional rigging of the hand.

FIGS. 3K and 3L are perspective views illustrating an embodimentincorporating a hand rig into the 3D mapped hand.

FIGS. 4A and 4B are perspective views illustrating the user's hand andfingers used to combine a gesture with a motion.

FIG. 4C is a perspective view illustrating the user using performing andselecting a gesture interface control function that involves multiplefingers.

FIG. 4D is a perspective view illustrating the user performing andselecting a gesture interface control function that involves one finger.

FIG. 4E is a perspective view illustrating the user performing andselecting a gesture interface control that involves touching a specifiedfinger to an identified point or area on the hand.

FIG. 4F is a perspective view illustrating the user performing andselecting a gesture interface control function that involves touchingone specified finger to another specified finger.

FIG. 5A is a perspective view illustrating a single point of control fora 2D or 3D computing environment on an external networked device using asingle finger as a controller.

FIG. 5B is a perspective view illustrating multiple points of controlfor a 2D or 3D computing environment on an external networked deviceusing multiple fingers as controllers.

FIG. 6A is a perspective view illustrating a wrist console projecting agraphic user interface (GUI) onto the user's hand and fingers.

FIG. 6B is a perspective view illustrating a user performing touch andgesture interfacing to control a projected GUI on the user's hand andfingers.

FIG. 6C is a perspective view illustrating a wrist console projecting agraphic user interface onto the users hands and fingers.

FIG. 6D is a perspective view illustrating the user typing on aprojected keypad on the users hand.

FIG. 7 is a perspective view illustrating a user typing on a projectedkeyboard mapped on to an external surface by both a left and right wristconsole

FIG. 8A is a perspective view illustrating the user interfacing with twonetworked devices and using a wrist console to map, image and model ascanned physical object into a virtual computing environment on anexternal networked device.

FIG. 8B is a perspective view illustrating the user operating a left andright wrist console as a two hand gesture interface controller tointerface and manipulate a 3D computer model scanned and modeled by thewrist console and wirelessly uploaded to an external networked device.

FIG. 8C is a perspective view illustrating the user selecting a file,document or program on one external networked device.

FIG. 8D is a perspective view illustrating the user operating andcontrolling a 3D scanned and modeled object on a wrist consoletouchscreen interface.

FIG. 9A is a perspective view illustrating a user wirelessly interfacingand controlling a remote device or vehicle.

FIG. 9B is a perspective view illustrating a user wirelessly sending andreceiving real-time voice, data, video and multimedia content to andfrom a remote device.

FIGS. 10A and 10B are perspective views illustrating a wrist consolefull body scanning, depth mapping and imaging process where a userperforms a body scan and a 3D computer model of the user is generated.

FIGS. 10C and 10D are perspective views illustrating the user indifferent positions and performing different body motions enabling thewrist console to map and image the body in multiple positions andanalyze the flexibility and mobility of the user to more accuratelygenerate the body rigging for the 3D computer model and replicate theusers motions in a virtual computing environment.

FIGS. 10E and 10F are perspective views illustrating the 3D mapping andimaging of the user and a 3D computer model with clothing.

11A and 11B are perspective views depicting the camera view from a topmodule of a wrist console.

FIGS. 11C and 11D are perspective views depicting the camera view from abottom module of a wrist console.

FIGS. 12A and 12B are perspective views depicting the camera view from atop module of a wrist console.

FIGS. 12C and 12D are perspective views depicting the camera view from abottom module of a wrist console.

FIGS. 13A-13C are perspective views illustrating the body riggingprocess.

FIGS. 14A and 14B are perspective views illustrating the users 3Dcomputer model spatial position and location in a mapped physicalenvironment.

FIG. 15 is a perspective view illustrating a virtual internal body mapof a user's anatomy and networked, sensors, implanted devices andprosthetics

FIGS. 16A and 16B are perspective views illustrating a wrist consolespace, object and environment 3D light and image mapping process.

FIGS. 17A and 17B are perspective views of the user standing in aresidential living space.

FIG. 18 is an overhead perspective view of the user and a wrist consoleidentifying and mapping the location, device type, functions andapplications, data, power and other system specifications and availablenetworks and interfacing options for all networked devices in the in theresidential living space.

FIGS. 19A and 19B are perspective views of a professional tennis playerwearing a wrist console.

FIGS. 20A, 20B, and 20C are perspective views of a professional golfermapping his swing, the ball, and virtualizing the player's entire gameof golf in real time.

FIG. 21 is a perspective view illustrating a professional golfer on agolf course.

FIG. 22 is a schematic diagram of a wrist console.

DETAILED DESCRIPTION

Apparatus and methods are described for autonomous, intelligent wearablewireless voice, data, and video communications systems that combineonboard computing with a multi-interface wrist console, remote deviceand graphic user interface (GUI) controller. Apparatus and methods arealso described for 3 Dimensional (3D) optical hand, body, object,sensor, and environment imaging, mapping, modeling, networking, andinterfacing. Further, apparatus and methods are described for enablingaugmentation of real world people, objects, and devices into a virtualenvironment and for enabling augmentation of virtual objects andinterfaces into a physical environment.

Apparatus and methods are described that are advancements in wearablecomputing optics and interfacing over prior art, expanding from hand,finger, and object positioning to a 3D scanning, mapping, modeling,imaging, projection, and wireless interfacing systems all captured,rendered, and operated by a wrist computing and control console.

Apparatus and methods are described implementing a combination of lightemitters; and sensors including body, motion, orientation, and locationsensors. These apparatus and methods may further include optic,stereoscopic, or plenoptic lens arrays to generate depth maps and 3Dimaging models, as well as virtual computing, interfacing and networkingplatforms by dynamically scanning and imaging the hands, body, objectsand environment, indoors or outdoors in daylight or at night using oneor more depth measurement and imaging methods.

Further, apparatus and methods are described for mobile light andoptical depth mapping imaging, modeling, networking, and interfacing onan autonomous, intelligent wearable wireless wrist computing, display,and control system for onboard and remote device and graphic userinterface (GUI) control. Augmentation of real world people, objects, anddevices into a virtual environment and augmentation of virtual objects,interfaces, and environments into the real world through wirelessmultimedia streaming and multi-interface display and projection systemis also described.

Light Mapping

Embodiments of the invention involve incorporating narrow or wide beamlight emitters, or a structured light imaging system on the top andbottom of wrist consoles worn by a user. In embodiments, a device scansthe top and bottom of the hand and fingers, the body, and any objects inthe hand or in the field of the light imaging system.

When using narrow or wide beam emitter and sensor arrays, the emitterarray may be assigned to move the focal point of the light beams up anddown and back and forth across an x and y axis grid formation during thescanning process and are then assigned fixed positions to monitor handand finger motion. The emitters form an x-y array for detecting the handand fingers in the x and y dimensions of space and the sensors detectthe presence or absence of a reflection. In embodiments, the depth (z)distance is measured by triangulation of the light beam reflection offof the target surface to the light sensor either mapping the targetsurface depth when the light beams are in motion and the object isstationary or dynamically identifying the position of the scanned objectwhen the light beams are fixed and the scanned object is in motion.

In embodiments using a structured light imaging system light emittersand diffusers are incorporated to produce a light speckle pattern acrossthe top and bottom of the hand and cover surrounding objects andenvironment. In embodiments designated camera sensors on a wrist consolerecognize the light speckle pattern and the directional reflection ofeach dot off of the target surface land on a different pixel within thecamera sensor to triangulate the beam of light and determine theposition, depth, and shape of the target surface.

Stationary structured light imaging system provides a constant point oforigin for the projected light pattern with the only variables being theposition, depth, and surface shape of target objects in its projectionfield. A wrist console introduces a wearable structured light imagingsystem that at most times is in motion. Even when a person holds theirarm and hand steady, slight body movements can alter both the positionand direction of the light emitters and projected light pattern andconsequently the position and line of sight of the camera sensors.

In embodiments, the light mapping system is used both for initialmapping and modeling of the hand, body, objects, and environment and todynamically monitor hand, finger, body, and object motion for gestureinterfacing and control and to perform instant keyless user verificationand authorization upon device sign-in, as well as performing instantuser verification for payment and other secure transactions and securityrelated functions such as keyless entry to home and vehicles and accessto unique user accounts and user specific functions and applications onthe device.

Position and Orientation Mapping

In embodiments, it is necessary to incorporate constant motion,position, and orientation data. When light and optical depth mapping and3D color imaging is performed, the spatial position, directional motion,and orientation of the wrist console is acquired by any combination ofonboard accelerometers, altimeters, compasses, and gyroscopes, as wellas GPS and radio frequency (RF) directional signal and location data tocontinuously identify the precise relational position of the wristconsole cameras, light emitters, and sensors to reflected and imagedsurfaces to assign that data to each light point and color pixel in adepth and color map.

Optical Mapping and Imaging

In embodiments, depth mapping and 3D imaging is achieved using astereoscopic or plenoptic multi-lens arrays. These arrays enable a wristconsole's top and bottom modules to dynamically capture 3D or 4Dmulti-depth of field color imaging of the hand, body, surroundingobjects, and environment.

In embodiments, when incorporating one or more stereoscopic lens arraysa wrist console performs stereo triangulation by determining the depthof two or more focal points in the scene, and determining the depths ofthe corresponding points in other images by matching points and featuresin one image to corresponding points and features in other images. Toovercome the correspondence problem, the stereoscopic imaging system mayselect to incorporate the light imaging system to project one or morepoints of light on a target surface enabling the imaging system toverify the precise corresponding points in the images. Once thecorresponding points have been identified, the imaging system determinesthe focal depths of all other points in the scene.

In embodiments, when incorporating a light-field plenoptic micro-lensarray the wrist console captures multiple depths of fieldsimultaneously. While stereoscopic lens arrays are limited to two ormore individual lens arrays and sensors, each capturing light and colorfrom a single depth of field, necessitating corresponding image analysisto match points in two or more images, the plenoptic micro-lens arrayassigns multiple lenses to a single sensor and captures the light andcolor from the entire field of view, while each lens captures adifferent depth of field enabling the camera to assign depth to allpoints in a captured image.

In embodiments, the optical imaging system is used both for initialimaging, depth and color mapping, and modeling of the hand, body,objects, and environment and to dynamically image hand, finger, body,and object motion for gesture and projection interfacing, to performuser verification and authorization and other security relatedfunctions, and to capture video and live stream user activities in 2Dand 3D or 4D video and perform other imaging applications.

Modeling and Rigging

After light scanning and 3D imaging an object, the corresponding depthmap is converted to a point cloud, a map of vertices with correspondingvectors in which each point is assigned an x, y and z (depth)coordinate. This process turns a grey scale depth map generated by thelight scanning process or a 3D imaging of an object into a vertex inwhich each point or pixel in the image is identified as an x, y, and zcoordinate that can be converted into metric units.

In embodiments, when a light scanned depth map is converted to a vertexand vector map by identifying the precise depth and directional positionof each surface point, the color mapping process is enabled in whichcorresponding depth mapped color pixels are assigned to each point onthe 3D vertex and vector map. This process converts the point cloud intoa mesh in which points on a contiguous surface are connected anddetermines, for example, that one finger is behind the other, and theyare not a single surface. The grid follows the surface shape, texture,and contours of the 3D mapped object.

Converting a surface mesh and 3D map of a persons hand or body into afunctional character model that can be animated to mirror the movementsof the wearer, incorporates a process of mapping the persons joints andassigning joint positions to the matching areas on the 3D model andgenerating an internal model rigging similar to the skeletal structurein the human body. Then attaching the rigging to the 3D mesh and modeland assigning areas of influence to the mesh and surface of the 3D modelsimilar to the effect of muscles and tendons on body motion and theskin.

In embodiments, when an existing functional character rigging exists,rather than generating a rig for each new model, the existing rig isscaled and conformed to the dimensions, shape, and physicalcharacteristics of the mapped person. This may incorporate a program fordetermining body flexibility and motion based on the body type, sex,size, weight, age, health, fitness, flexibility, and other parameters ofthe mapped person to more accurately conform the rig and model to mimicthe natural body motion and mobility of the person.

During the 3D light mapping and imaging process the wrist console mayprompt the wearer to perform a number of hand and body motions,gestures, and positions to identify the joints, bone structure andmobility of the person. This may necessitate capturing multiple 3D scansand images of the person and then adjusting the rigging to replicate theprecise body structure and mobility.

Sensor Mapping and Interfacing

In embodiments a wrist console is used to continuously map full bodymotion in real-time and incorporates external sensors into its 3Dmapping and interfacing system. This includes body, clothing and remotewireless equipment sensors. By attaching micro sensors to the body orclothing on each of the limbs and joints or networking with embeddedsensors in clothing, shoes and equipment the wrist console can identifythe spatial position of one or more wireless sensors and assign thosesensors to the mapped 3D model of the person, equipment and environment.

In embodiments the wrist console may use one or a combination ofnetworking methods including Radio Frequency (RF), light/IR, Near FieldCommunication (NFC), Bluetooth, WiFi and Cellular networks for local andremote sensors and devices interfacing and control. This sensor networkenables both sending and receiving data by the wrist console forwireless operation and control of remote sensors and dynamic mapping,interfacing and streaming of networked data as well as onboard or remotestorage of mapped data.

In embodiments when the wrist console operates as a sensor hub for awireless sensor network (WSN), the wrist console networks with eachsensor directly or via a mesh network in which each sensor operates as anode and not only captures and sends data but also serves as a relaypassing data on to the other nodes in the network.

In embodiments when monitoring body motion by identifying the 3Dposition, velocity, and acceleration of each joint or body part, acomplete Cartesian coordinate 3D model of the body may be describedmathematically with distance coordinates of x, y, and z; velocitycoordinates of v_(x), v_(y), and v_(z); and acceleration coordinates ofa_(x), a_(y), and a_(z) to calculate the future position of an object inmotion. Once the wrist console has identified and networked with theindividual sensors or mesh sensor group, the wrist console is able tomap the precise position of the sensors on the 3D character model. Thisprocess enables the wrist console to capture full body motion andacceleration as a continuous data stream and assign that data to the 3Drigged virtual model of the wearer to provide a real-time animation ofthe body and full body interfacing in a virtual environment.

In embodiments when the wrist console is used to map the internal bodyanatomy and interface with internal body sensors, devices andprosthetics, the wrist console incorporates a similar method of mapping,modeling, networking and interfacing with the internal body as it doeswith the external body. In embodiments when the wrist console is mappingthe external body using the light and optical mapping and externalsensors the wrist console is also performing internal mapping whichincorporate the wrist console's onboard health and body sensors and thenexpands to all networked internal body sensors including ingested andimplanted sensors, devices, prosthetics and any body or brain machineinterfacing systems.

In embodiments the wrist console incorporates onboard body health andfitness sensors including top and bottom module wrist facing Infrared(IR) spectroscopy and pulse oximeter, heart rate monitor, thermometer,galvanic response system, Electroencephalograph (EEG),Electrocardiograph (ECG), Electromyograph (EMG), and glucose meter.

Projection Mapping and Interfacing

In embodiments, incorporating a pico projector for projecting an imageonto external surfaces, the light and image mapping systems andorientation system are used to depth map surfaces, dynamically map thespatial position of the hands and fingers and the relational position ofthe wrist console and projector to a target surface. These processesenable the wrist console to map a projected display and graphic userinterface onto any surface. The light and optical mapping systems arealso used to dynamically monitor hand and finger motions and gesturesenabling the user to perform touch and gesture interfacing to controlthe projected interface.

Further embodiments include an active touch screen displays,microphones, speakers, tactile feedback (haptic) sensor arrays, andfront facing video cameras. These embodiments enable touch, voice, andgesture interfacing and voice command, video conferencing, and dynamictouch screen display with onboard graphic user interfaces.

In embodiments, the wrist console incorporates a touch screen display,one or more microphones and speakers, and tactile feedback (haptic)sensor array. These embodiments provide touch, voice, and gestureinterfacing options for the user, enabling the user to select the mosteffective method for displaying and interfacing with a graphic userinterface either on the wrist console or on one or more networkeddevices.

In embodiments, the user can map or assign a specific user interfacesuch as voice command or gesture interfacing to a specific function,application, or device. For example, if the user is using the wristconsole to interfacing with a personal computer and a television theuser may assign voice command to the television while using gesture andtouch on the computer.

In embodiments, the wrist console incorporates haptic sensor stripsand/or a ring of haptic sensors on the inside of both the top and bottomwrist units. The wrist console generates very intricate positional,vibrational, and pressure responses to minute finger, hand, wrist, andbody movements. The tactile response may also be incorporated intogesture command, touch screen, and device controls and other userinterface applications to simulate button press on a projected keyboard,or provide a tactile response and more realism to object and/orapplication selection and control in a virtual 2D or 3D environment. Thehaptic response system may also be used to indicate an incoming oroutgoing call, text or other event, locational and/or relationaldistance to a recognized object or person or any other assignedcontextual application, alarm or monitored health status event such asalerting the wearer when their heart rate rises above a designated rateor glucose levels fall above or below a designated level or to informthe wearer of a potential oncoming seizure. Different types ofvibrational and/or electro-stimulated responses may be generated andassigned to different callers, events and applications.

Device Mapping and Interfacing

In embodiments, the wrist console is capable of streaming content thatis stored and playing on the device and or streaming to the wristcontroller from the Internet to the screens of one or more networkeddevices and/or streaming multimedia content from a networked TV, gameconsole, PC, or other networked device to one or more other devices ordisplays. This peer-to-peer networking, content management,distribution, and streaming can be achieved using a number of differentwireless networks. Some of those include WiFi, Bluetooth, cellular,Infrared/light, Radio Frequency (RF), and NFC for rapid payments andtransactions. One method for connecting all displays and devices in itsfield of view is through a single WiFi peer-to-peer network where eachdevice is connected wirelessly through a multi-channel WiFi directconnect platform operating as a standalone WiFi hotspot and router, thewrist controller creates an ad-hoc peer-to-peer network with one or morewireless and/or Internet enabled devices and operates as remote wearablevideo game and computing console and wireless hub. The wrist console mayalso use any combination of networks to communicate with one or moredevices

In embodiments, the wrist console manages content across multiplenetworked devices and monitors based on the position of the display inthe room and the relation of the display to the wrist console and user.The wrist console is able to connect with multiple devices usingmultiple methods, networks, and channels.

Detailed Overview of the Embodiments in the Drawings

FIG. 1A is a perspective view of an embodiment of the wrist console 101depicting the top wrist module 102 and bottom wrist module 103 whichserve as housing modules for internal components of the device. Thewrist modules are connected with adjustable wrist straps 104 which inembodiments contain communication cables between the two wrist modules.The device further has a wrist strap release and locking system 105,forward facing light emitters 106 and sensors 107, and multi-camera lensarray 108. FIG. 1A shows the inside view of the bottom module bodysensors including light emitters and sensors 109, Galvanic Skin ResponseSystem (GSR) 110, and haptic feedback arrays (haptic array) 111, apartial view of the display 114, microphones 115, speakers 116, and topfacing cameras 117.

FIG. 1B is a perspective view of the embodiment shown in FIG. 1Adepicting a view of the top module 102 body sensors including lightemitters and sensors 109, GSR 110 and Haptic Array 111.

FIG. 1C is a perspective view of the embodiment shown in FIG. 1Adepicting an overhead view of the rear facing light emitters 106 andsensors 107, rear multi-camera lens array 108, power and data ports 112and docking ports 113.

FIG. 1D is a top plan view of the embodiment shown in FIG. 1A depictingthe display 114, microphones 115, speakers 116 and top facing cameras117.

FIG. 2A is a perspective view of the embodiment shown in FIG. 1Adepicting an expansion module 201 with display 202 and release buttons203, attached to the top module 102 of the wrist console 101. Thisexpansion module may serve as an additional power and data storage,processing and/or communication system for the device and/or an expandeddisplay and interfacing system and may also perform expanded servicessuch as a plug in glucose meter or other application.

FIG. 2B is a perspective view of the embodiment shown in FIG. 1Adepicting the detached expansion module 201 with release buttons dockingtabs 204 and power and data plug 204 and a button array 206 on the rearof the top module 102.

FIG. 2C is a perspective view of the embodiment shown in FIG. 1Adepicting an expansion module 201 attached to the bottom module 103.

FIG. 2D is a perspective view of the embodiment shown in FIG. 1A with anexpansion module 201 detached from the bottom module 103.

FIGS. 3A-3L are perspective views illustrating a wrist console 101performing depth mapping and 3D imaging the hand and fingers 301,identifying the joints and then rigging a fully functional computermodel of the hand (hand model) 307.

FIGS. 3A and 3B are perspective views illustrating an embodiment of thewrist console 101 that incorporates one or more moving beams of light302 performing a light scan of the top and bottom of the hand andfingers 301 as a method for depth mapping. Triangulation is determinedby the wrist console 101 light emitters 104 and sensors 106 as the lightbeams 302 move vertically and horizontally across the face of both sidesof the hand and fingers 301.

FIGS. 3C and 3D are perspective views depicting another embodiment ofthe invention that incorporates structured light imaging 303 into itsdepth mapping process by illuminating the top and bottom of the hand andfingers 301 with a speckled light pattern to light map 303 the entirehand 301 at once.

FIGS. 3E and 3F are perspective views illustrating the wrist console 101generating a depth and color mapped hand 304 performing a combination oflight mapping and imaging using 3D cameras 108 or 4D imaging using aplenoptic multi-lens array cameras 108 on both the top module and bottomof the wrist console 101.

FIGS. 3G and 3H are perspective views illustrating the wrist console 101identifying the precise position of joints and creases 305 on the topand bottom of the hand and fingers 301 for the purpose of generating arig for the 3D mapped hand 304.

FIGS. 3I and 3J are perspective views illustrating the wrist console 101generating a functional rigging of the hand (hand rig) 306 from the topand bottom perspective.

FIGS. 3K and 3L are perspective views illustrating the inventionincorporating the hand rig 306 into the 3D mapped hand 304 to create afully functional rigged computer model of the hand (hand model) 307capable of being animated and replicating the movements of the usershand and fingers 301 in real-time.

FIGS. 4A-4F and FIGS. 5A and 5B are perspective views illustrating anapplication of the invention where the user performs and selects controlfunctions for different gestures. The wrist console 101 assigns thosecontrols to the hand model 307, which is used to carry out the usersgesture inputs and commands in a 2D or 3D computing environment. FIGS.4A-4F and FIGS. 5A and 5B represent only a few examples of gestureinterfacing control options for a potentially limitless custom interfaceprogramming system.

FIGS. 4A and 4B are perspective views illustrating the user's hand andfingers 301 used to combine a gesture (making a first) with a motion(turning and moving the wrist) to a assign a customized gestureinterface control function. The gesture is mapped and recorded by thewrist console 101.

FIG. 4C is a perspective view illustrating the user using performing andselecting a gesture interface input and control function that involvesmultiple fingers 301.

FIG. 4D is a perspective view illustrating the user performing andselecting a gesture interface control function that involves one finger301.

FIG. 4E is a perspective view illustrating the user performing andselecting a gesture interface control that involves touching a specifiedfinger to an identified point or area on the hand 301.

FIG. 4F is a perspective view illustrating the user performing andselecting a gesture interface control function that involves touchingone specified finger to another specified finger 301.

FIG. 5A is a perspective view illustrating a single point of control fora 2D or 3D computing environment on an external networked device 802using a single finger as a controller 301.

FIG. 5B is a perspective view illustrating multiple points of controlfor a 2D or 3D computing environment on an external networked device 802using multiple fingers as controllers 301.

FIG. 6A is a perspective view illustrating the wrist console 101projecting a graphic user interface (GUI) 601 on to the user's hand andfingers 301. The projected interface is mapped onto the hand using thelight mapping and 3D imaging system.

FIG. 6B is a perspective view illustrating a user performing touch andgesture interfacing to control a projected GUI 601 on the user's handand fingers 301.

FIG. 6C is a perspective view illustrating the wrist console 101projecting a graphic user interface 601 onto the users hands andfingers. In FIG. 6C the user has spread out their hand and fingers andthe projected interface has dynamically conformed to the new position ofthe hand and fingers 301. The user is selecting one of the projectedicons representing an active program or application running on wristconsole 101 or remotely via an Internet connection.

FIG. 6D is a perspective view illustrating the user typing on aprojected keypad 601 on the users hand 301.

FIG. 7 is a perspective view illustrating a user typing on a projectedkeyboard 701 mapped on to an external surface by both a left and rightwrist console 101. FIG. 7 depicts an embodiment with a coordinated dualprojected interface in which both left and right wrist consoles operatein concert in mapping and projecting a dynamic interface on a projectedsurface. FIG. 7 depicts the user typing a document on the projectedkeyboard that is displayed on an external device 803. The left and rightwrist consoles are either operating as a single input device andrelaying data wirelessly to a remote control device 803 or the wristconsoles 101 are operating as primary operating and control device andstreaming data to a remote display 803.

Figure Sets 8A-8D and 9A and 9B are perspective views depicting thewrist console wirelessly interfacing with external devices. In each ofthe figures the wrist console 101 is shown on both wrists, although apair of consoles may be operated as a single device or device pair, eachwrist console 101 may also operate autonomously and does not need asecond console to perform two handed gesture interface control. A singlewrist console 101 is capable of monitoring a second hand in closeproximity for dual hand interfacing or may operate in concert with asecond wrist console 101 enabling expanded functionality such asmulti-function two-handed control, dual projection, expanded networking,processing, interfacing, power and data storage

FIG. 8A is a perspective view illustrating the user interfacing with twonetworked devices and using the wrist console 101 to map, image andmodel a scanned physical object into a virtual computing environment onan external networked device 802.

FIG. 8B is a perspective view illustrating the user operating a left andright wrist console 101 as a two hand gesture interface controller tointerface and manipulate a 3D computer model scanned and modeled by thewrist console 101 and wirelessly uploaded to an external networkeddevice 802.

FIG. 8C is a perspective view illustrating the user selecting a file,document or program on one external networked device 802 and with agesture, voice or other UI command wirelessly transferring the file,document or program to a second networked device 803 using the wristconsole as a data bridge between two networked devices.

FIG. 8D is a perspective view illustrating the user operating andcontrolling a 3D scanned and modeled object on the wrist console 101touchscreen interface.

FIG. 9A is a perspective view illustrating a user wirelessly interfacingand controlling a remote device or vehicle 901 using a wide or localarea peer-to-peer wireless network or via the Internet using a wide orlocal area Internet connection.

FIG. 9B is a perspective view illustrating a user wirelessly sending andreceiving real-time voice, data, video and multimedia content to andfrom a remote device 901, streaming the data and multimedia content in3D to a left 904, and right 903, binocular heads up display 902.

FIGS. 10A and 10B are perspective views illustrating the wrist console101 full body scanning, depth mapping and imaging process where a user1001 performs a body scan and a 3D computer model 1002 of the user isgenerated.

FIGS. 10C and 10D are perspective views illustrating the user indifferent positions and performing different body motions enabling thewrist console 101 to map and image the body 1002 in multiple positionsand analyze the flexibility and mobility of the user to more accuratelygenerate the body rigging for the 3D computer model and replicate theusers motions in a virtual computing environment.

FIGS. 10E and 10F are perspective views illustrating the 3D mapping andimaging of the user 1001 and the 3D computer model 1003 with clothing.This may be accomplished by light mapping and 3D imaging the user's 1001physical clothing or by mapping virtual clothing onto the 3D model 1003in a computing environment.

FIGS. 11A-11D are perspective views illustrating the body mapping andimaging process from the perspective of each of the wrist console's 101body facing cameras 108 with FIGS. 11A and 11B depicting the camera viewfrom the top module and FIGS. 11C and 11D depicting the camera view fromthe bottom module 103. The wrist console 101 is not shown in FIGS.11A-11D because the figures depict the perspective of the cameras.

FIGS. 12A-12D are perspective views illustrating the body mapping andimaging process from the perspective of each of the wrist consoles 101body facing cameras 108 with FIGS. 12A and 12B depicting the camera viewfrom the top module 102 and FIGS. 12C and 12D depicting the camera viewfrom the bottom module 103. In FIGS. 12A-12D the users arms and handsare stretched out higher over the users head enabling the cameras toscan a different portion of the users body.

FIGS. 13A-13C are perspective views illustrating the body riggingprocess with FIG. 13A illustrates the surface mesh of depth and colormapped model of the user 1002. FIG. 13B illustrates a full bodycharacter rigging (rig) that is conformed to the precise dimensions andcharacteristics of the mapped computer model of the user 1002. And FIG.13C illustrates the incorporation of the character rig

FIGS. 14A and 14B are perspective views illustrating the user's 1001 andthe users 3D computer model 1002 spatial position and location in amapped physical environment 1301 identified and mapped during the 3Dbody mapping and imaging process.

FIG. 15 is a perspective view illustrating a virtual internal body map1201 of the users anatomy and all networked, sensors, implanted devicesand prosthetics all mapped and wirelessly controlled by the wristconsole 101. In FIG. 15 the wrist console 101 using onboard, external,implanted or ingested networked body sensors to map each of the usersbody systems; Nervous System 1202, Endocrine System 1203, SkeletalSystem 1207, Muscular System 1208, Integumentary System 1209,Cardiovascular System 1210, Respiratory System 1211, Lymphatic System1212, Digestive System 1213, Urinary System 1214 and Reproductive System1215. The wrist console 101 also networks and interfaces with allinternal data and multimedia interfacing systems, depicted in FIG. 15 asa Brain Machine Interface (BMI) 1204, Prosthetics depicted in FIG. 15 asa prosthetic eye 1205, and other implanted devices depicted in FIG. 15as a pacemaker 1206.

FIGS. 16A and 16B are perspective views illustrating the wrist console101 space, object and environment 3D light and image mapping processshown in FIGS. 16A and 16B as a residential living space 1301. FIG. 16Adepicts the user 1001 3D mapping and imaging the living and diningsection 1306 while FIG. 16B depicts the user 1001 mapping the kitchensection 1304 of the residential living space 1301.

FIGS. 17A and 17B are perspective views of the user standing in aresidential living space 1301. FIG. 17A illustrates the user standing inthe physical residential living space 1301 providing an example of apotential 3D mapped environment. FIG. 17B illustrates an overheadperspective view of a 3D mapped user 1003 and environment 1302 with allmapped people, objects, devices and environments stored securely on thewrist console or uploaded wirelessly to a user authorized account on theInternet or other network or database.

FIG. 18 is an overhead perspective view of the user 1001 and the wristconsole 101 identifying and mapping the location, device type, functionsand applications, data, power and other system specifications andavailable networks and interfacing options for all networked devices inthe in the residential living space 1301.

FIGS. 19A and 19B are perspective views of a professional tennis player1401, wearing the wrist console 101, playing tennis on a real outdoortennis court 1404 while the user 1001 is testing his skills at home byattempting to return the tennis ball 1403 hit by the tennis player inreal-time in a virtual gaming environment on the users 1001 televisionor other display 1405.

FIGS. 20A-20C are perspective views of a professional golfer 1501mapping his swing, the ball 1504, and virtualizing the players entiregame of golf in real time. FIG. 20A is a perspective view illustrating aprofessional golfer swinging at a golf ball 1504. The golfer has sensorson or embedded in his clothing and shoes 1502 and equipment 1503enabling the wrist console 101 to map every detail of the golfers bodymotion during the swing.

FIG. 21 is a perspective view illustrating the professional golfer 1501on the golf course mapping his swing and remotely mapping and monitoringthe height, speed, trajectory, landing and resting position of a sensorenabled networked golf ball on the wrist console 101.

FIG. 22 is a schematic diagram of the connections of various componentsenvisioned to be part of a wrist console device.

Although the invention is described herein with reference to thepreferred embodiment, one skilled in the art will readily appreciatethat other applications may be substituted for those set forth hereinwithout departing from the spirit and scope of the present invention.Accordingly, the invention should only be limited by the Claims includedbelow.

1. An apparatus, comprising: a 2 Dimensional (2D), 3 Dimensional (3D),or 4 Dimensional (4D) imaging, multi lens array wrist mounted computing,communications, and control device, including a light mapping, scanning,and projection system, said device comprising: a housing module housinga processor, light emitters, and optical sensors; wherein said processoris in communication with said light emitters and said optical sensors;and wherein said light emitters and optical sensors function with saidprocessor to scan surfaces of a user's body and create maps of scannedbody surfaces.
 2. The apparatus of claim 1, wherein said optical sensorscomprise a camera.
 3. The apparatus of claim 2, wherein said camerafunctions to scan objects within the close proximity of a user; andwherein said processor processes data received from said camera tocreate a map of a scanned object.
 4. The apparatus of claim 3, whereinsaid camera functions to scan multiple objects and surfaces within closeproximity of the user; and wherein said processor processes datereceived from said one or more cameras to create a map of theenvironment around the user.
 5. The apparatus of claim 4, furthercomprising: a location determination module capable of determining theprecise location of the wristed mounted computing device; wherein saidprocessor processes date received from the location determination modulein connection with data received from said camera to create the map ofthe environment around the user.
 6. The apparatus of claim 1, furthercomprising: a signal receiving module for receiving data and positionsignals from sensors mounted on body surfaces of the user, within thebody of the user, or on external objects.
 7. The apparatus of claim 1,further comprising: an infrared emitter and sensing module directedtoward the inside of the wrist and functioning to detect the heart rateof the user, the oxygen content of the user's blood, or tendon movementwithin the user's wrist.
 8. The apparatus of claim 1, furthercomprising: a projector for projecting a graphical user interface on asurface; wherein the processor determines a user's interaction with aprojected user interface with the map of the body surface.
 9. A methodof interfacing a wrist mounted computing devices with another device,comprising: mapping a portion of the body of a user with a wrist mountedcomputing device by: scanning a surface of the body of a user, andprocessing data received in the scanning step within a processor in thewrist mounted computing device to create a map of the surface of thebody scanned; and interfacing with an external device using the map ofthe surface of the body.
 10. The method of claim 9, wherein interfacingwith an external device comprises: projecting a graphical user interfaceonto a surface with a projector within the wrist mounted computingdevice; and determining user interaction with the graphical userinterface using a map of the surface of the body created by theprocessor.
 11. The method of claim 9, wherein interfacing with anexternal device comprises: determining gestures performed by user usinga map of the surface of the body scanned created by the processor; andtransmitting predetermined signals to an external device based on thedetermined gesture.
 12. The method of claim 9, wherein interfacing withan external device comprises: determining the location of the wristedmounted computing device using a position determination module; anddetermining a specific device out of multiple possible devices tointerface with based on the location of the wristed mounted computingdevice and the map of the surface of the body scanned created by theprocessor.
 13. The method of claim 12, wherein determining the locationof the wristed mounted computing device further comprises: scanningsurfaces and objects in close proximity to the user; and processing datareceived in the scanning surfaces and objects step within a processor inthe wrist mounted computer device to create a map of the environmentaround the user.